Current Rucio app status:
Functional tests are back online.
Some issues around the new oidc token workflows required bodges. Running a custom daemon image with following changes:
rse.py ln 1849 (hardcoded WLCG /any audience rather than the fts hostname) > this requires further liaising with Rose, there is a configuration item in FTS that restricts audiences and the new version sets the fts token audience to the fts hostname. rse.py ln 1876 (hardcoded scopes for storage.read:/ and storage.modify:/ > issues with some storage technologies highlighted by James W, sounded like it could be fixed RSE side ultimately but workaround for now. The new determine_scope functions add the full path on to these claims, but some storage techs don't expect the base fts3.py ln 873 (hardcoded oidc_support == True otherwise poller doesn't attempt to get token) > need to talk to Dimitrios (Rucio dev) about this
On helm chart side:
https://github.com/rucio/helm-charts/blob/master/charts/rucio-daemons/values.yaml#L543 causes us problems as the usercert propagates into the config and conveyor daemons end up trying to get an x509 proxy. Can't just set this to empty in overridden values as that breaks it too. Needs a PR raising to remove this as a default in the first place. As the logic for user certs doesn't get called if oidc_support == True, a modified chart is currently not required with the bodge to fts3.py in place.
Functional testing matrix looks bleak. Suspect issues with token scopes and audiences as there's new logic for determining these per RSE:
For conveyor, request_token called from:
https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L911 https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L1032 https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L1038
and reaper:
https://github.com/rucio/rucio/blob/master/lib/rucio/daemons/reaper/reaper.py#L623
and the request_token function itself:
https://github.com/rucio/rucio/blob/master/lib/rucio/core/oidc.py#L125
Note that the determination of rse scope/audience is in core/rse.py.
Current Rucio migration status:
Expanded remit to include also altering the configuration variables.
As an example:
Old env var: RUCIO_CFG_AUTH_TYPE Corresponding new env var: RUCIO_CFG_CLIENT_AUTH_TYPE
This applies for all config attributes under the [client] section of the rucio.cfg file.
Done:
Services left:
Would be wise to scan through gitlab repo for any other bits.
! The previous modified policy package also needs bringing along.
Conveyor issues:
fts audience needs to be hostname - > fts3-ska.scd.rl.ac.uk not "fts" *submitter, hardcoded for now to "fts https://wlcg.cern.ch/jwt/v1/any" https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L911
usercert in rucio cfg in poller? coming in from somewhere, not part of config? can't just leave empty str, have to remove the entire entry https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L175 > NEED TO RAISE PR but overriden if oidc_support = True anyway
also conveyor poller doesn't by default get token via oidc? need to make: https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L874 oidc_support = true and https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L911 (as before w/audiences)
Created custom image for this.
Now need to change all the envvars from RUCIO_CFG to RUCIO_CFG_CLIENT. Task manager is reenabled (but not working).
separate idpsecrets.json for server authserver and daemons, old user auth flow expects all entries to have old format but new daemons flow expects new format ( id , secret, issuer)
Everything but daemons work. New domain is rucio.srcnet.skao.int.
Databases migrated back to cinder, upgraded to 35.0.0
Did a metadata database version upgrade too since it was running an old version of pg
Daemons still getting stuck, they can't find the db.
The old server is "off"
Velero, manila, cinder all migrated.
ArgoCD deployed at argocd.srcnet.skao.int.
Rucio being migrated. Note, Manila can't snapshot shares so will be migrating back to Cinder so we can backup via openstack snapshots through Velero.
Next step is to export and reimport the databases into the corresponding locations on the new cluster.